So after quite a while I again had the opportunity to do the ftpmaster works of a Debian release. Not quite my first and hopefully not my last one, lets see. And, like last time, Mark joined to help with the work. Again, a few moments and not a detailed log (times in UTC):

• [08:30] I ll start the ftpmaster side by turning off all cronjobs, prepare some database backup, get myself sorted.
• [09:33] Got dak.git prepared with most of the changes we need for the release
• [10:02] Oldstable suite created, load the contents from stable into oldstable. Lenny is now oldstable
• [11:00] Load stable with the contents of testing. That is, stable now is squeeze.
• [11:35] Cleanup various dirs in dists/, like - old installers, some breakage from elsewhere, fixup READMEs.
• [11:52] Have Lenny Release file regenerated and signed
• [12:29] Have Squeeze Release file generated and signed
• [12:54] Push the cd builder machine so they can start building our CD/DVD images
• [13:00] Start a manual dinstall run to check through all the rest of the archive if it still works
• [13:43] Get a heart attack when clean-suites reports Cleaned 2543 files, 2500 MB , but then discover it is only the hppa removal that got in just late before the release. Puuh.
• [14:30] all of the manual dinstall run is done, everything fixed, the main archive looks nice. So lets start breaking security
• [15:40] Yay, security archive all done, waiting for the general release to push the world
• [17:30] After a short break I updated the backports.debian.org archive also, it is now ready to receive uploads targetting squeeze (but wait for a mail from the bpo team!), the lenny-backports suite is changed to need manual approval of uploads, all nice.

Right now (that is, 21:38 UTC) I am mosty waiting for the rest of the people working on the release (CD builders, press, website, ) to finish their task (building hundreds of images just takes ages). When that is done we can then do a coordinated push to the world. Last plans for that would mean in about an hour . Lets see, Mr. Murphy sure can be around. Thanks go out to everyone involved in preparing this release, be it in the past by fixing bugs, uploading packages, doing whatever was needed, as well as doing the work today. Update: If you miss us setting the new testing AKA wheezy to something inside there - there is no need to. It is simply the continutation of testing with just another name, which did not get changed contents, so nothing done for that. :)

For some reason, I don t really know why, I decided to switch my keyboard layout over to something weird. No, not the french brainfuck :) that is actually possible to type on in a halfway acceptable speed (shutup Bubulle), but the Neo thing for german keyboards. Hell. I didn t know how slow one can be typing. About a letter in 3 seconds, or something like it. I think, we should add something to our mandantory set of rules for anyone on any Debian Mailinglist: Whenever a flamewar starts, everyone who participates has to switch their keyboard layout to a random new layout. For every paragraph of every mail they write. Would stop the thread before it really happens. Ohwell, lets see where I stand in a week/month from now. Maybe I can actually use it then. And maybe I really type with 10 fingers then, and not something like the 3 til 8 i currenly use (depends on moon phase how many, i think). I m not slow with my current system, far from it, but I do sometimes have pain in my hands. Maybe that goes away too if I adopt something better, with less weird hand/finger movings. Though the way over will be painful in itself.

In response to another post I think the comic says it all best. :) (Not using garfield.com directly, as that useless thing of a page isn t usable. Flash ridden fuckup.

Debian s testing distribution is where Debian developers prepare the next stable distribution. While this is still its main purpose, many users have adopted this version of Debian because it offers them a good trade-off between stability and freshness. But there are downsides to using this distribution and the Constantly Usable Testing (CUT) project aims to resolve those. This article will present the project and the challenges involved to make it happen. About Debian unstable & testing Debian unstable is the distribution where developers upload new versions of their packages. It happens frequently that some packages are not installable due to changes in other packages or due to transitions not yet completed. Debian testing, on the contrary, is managed by a tool that ensures the consistency of the whole distribution: it picks updates from unstable only if the package has been enough tested (10 days usually), if it s free of new release-critical bugs, if it s available on all supported architectures, and if it doesn t break any other package already present in testing. The Release Team (RT) controls this tool and provide hints to help it find a set of packages that can flow from unstable to testing. Those rules also ensure that the packages that flow into testing are reasonably free of show-stopper bugs (like a system that doesn t boot, or X that doesn t work at all). This makes it very attractive to users who like to regularly get new upstream versions of their software without dealing with the biggest problems associated to them. This is all very attractive, yet several Debian developers advise people to not use testing. Why is that? Known problems with testing Disappearing software The release team use this distribution to prepare the next stable release and from time to time they remove packages from it. Either because it s needed to ensure that other packages can migrate from unstable to testing, or because they have long-standing release-critical bugs without progress towards a resolution. It also happens that they remove packages on request of the maintainers because they believe that the current version of the software cannot be supported (security-wise) for 2 years or more. The security team also regularly issues such requests. Long delays for security and important fixes Despite the 10-day delay in unstable, there are always some annoying bugs (and security bugs are no exceptions) that are only discovered when the package already has migrated to testing. The maintainer might be quick to upload a fixed package in unstable, and might even raise the urgency to allow the package to migrate sooner, but if the packages gets entangled in a large ongoing transition, it will not migrate before the transition is completed. Sometimes it can take weeks for that to happen. This delay can be avoided by doing direct uploads to testing (through testing-proposed-updates) but this is almost never used, except during a freeze, where targeted bugfixes are the norm. Not always installable With testing evolving daily, updates sometimes break the last installation images available (in particular netboot images that get everything from the network). The debian-installer (d-i) packages are usually quickly fixed but they don t move to testing automatically because the new combination of d-i packages has not necessarily been validated yet. Colin Watson sums up the problem:

Getting new installer code into testing takes too long, and problems remain unfixed in testing for too long. [...] The problem with d-i development at the moment is more that we re very slow at producing new d-i *releases*. [...] Your choices right now are to work with stable (too old), testing (would be nice except for the way sometimes it breaks and then it tends to take a week to fix anything), unstable (breaks all the time).

CUT s history CUT finds its root in an old proposal by Joey Hess: it introduces the idea that the stable release is not Debian s sole product and that testing could become with some work a suitable choice for end-users. Nobody took on that work and there was no visible progress in the last 3 years. But recently Joey brought up CUT again on the debian-devel mailing list and Stefano Zacchiroli (the Debian project leader) challenged him to setup a BoF on CUT for Debconf10. It turned out to be one of the most heavily attended BoF (video recording is here), there is clearly a lot of interest in the topic. There s now a dedicated wiki and an Alioth project with a mailing list. The rest of this article tries to summarize the various options discussed and how they re supposed to address the problems identified. The ideas behind CUT Among all the ideas, there are two main approaches that have been discussed. The first is to regularly snapshot testing at points where it is known to work reasonably well (those snapshots would be named cut ). The second is to build an improved testing distribution tailored to the needs of users who want a working distribution with daily updates, its name would be rolling . Regular snapshots of testing There s general agreement that regular snapshots of testing are required: it s the only way to ensure that the generated installation media will continue to work until the next snapshot. If tests of the snapshot do not reveal any major problem, then it becomes the latest cut . For clarity, the official codename would be date based: e.g. cut-2010-09 would be the cut taken during September 2010. While the frequency has not been fixed yet, the goal is clearly to be on the aggressive side: at the very least every 6 months, but every month has been suggested as well. In order to reach a decision, many aspects have to be balanced. One of them (and possibly the most important) is the security support. Given that the security team is already overworked, it s difficult to put more work on their shoulders by declaring that cuts will be supported like any stable release. No official security support sounds bad but it s not necessarily so problematic as one might imagine. Testing s security record is generally better than stable s one (see the security tracker) because fixes flow in naturally with new upstream versions. Stable still get fixes for very important security issues sooner than testing, but on the whole there are less known security-related problems in testing than in stable. Since it s only a question of time until the fixed version comes naturally from upstream, more frequent cut releases means that users get security fixes sooner. But Stefan Fritsch, who used to be involved in the Debian testing security team, has also experienced the downside for anyone who tries to contribute security updates:

The updates to testing-security usually stay useful only for a few weeks, until a fixed version migrates from unstable. In stable, the updates stay around for a few years, which gives a higher motivation to spend time on preparing them.

So if it s difficult to form a dedicated security team, the work of providing security updates comes back to the package maintainer. They are usually quite quick to upload fixed packages in unstable but tend to not monitor whether the packages migrate to testing. They can t be blamed because testing was created to prepare the next stable release and there is thus no urgency to get the fix in as long as it makes it before the release. CUT can help in this regard precisely because it changes this assumption: there will be users of the testing packages and they deserve to get security fixes much like the stable users. Another aspect to consider when picking a release frequency is the amount of associated work that comes with any official release: testing upgrades from the previous version, writing release notes and preparing installation images. It seems difficult to do this every month. With this frequency it s also impossible to have a new major kernel release for each cut (since they tend to come out only every 2 to 3 months) and the new hardware support that it brings is something worthwhile to many users. In summary, regular snapshots address the not always installable problem and changes the perception of maintainers towards testing, so that hopefully they care more of security updates in that distribution (and in cuts). But they do not solve the problem of disappearing packages. Something else is needed to fix that problem. A new rolling distribution? Lucas Nussbaum pointed out that regular snapshots of Debian is not really a new concept:

How would this differentiate from other distributions doing 6-month release cycles, and in particular Ubuntu, which can already be seen as Debian snapshots (+ added value)?

In Lucas s eyes, CUT becomes interesting if it can provide a rolling distribution (like testing) with a constant flux of new upstream releases . For him, that would be something quite unique in the Free Software world . The snapshots would be used as starting point for the initial installation, but the installed system would point to the rolling distribution and users would then upgrade as often as they want. In this scenario, security support for the snapshots is not so important, what matters is the state of the rolling distribution. If testing were used as the rolling distribution, the problem of disappearing packages would not be fixed. That s why there have been discussions of introducing a new distribution named rolling that would work like testing but with adapted rules, and the cuts would then be snapshots of rolling instead of testing. The basic proposal is to make a copy of testing and to re-add the packages which have been removed because they are not suited for a long term release while they are perfectly acceptable for a constantly updated release (the most recent example being Chromium). Then it s possible to go one step further: during freeze, testing is no longer automatically updated which makes it inappropriate to feed the rolling distribution. That s why rolling would be reconfigured to grab updates from unstable (but using the same rules than testing). Given the frequent releases, it s likely that only a subset of architectures would be officially supported. This is not a real problem because the users who want bleeding edge software tends to be desktop users on mainly i386/amd64 (and maybe armel for tablets and similar mobile products). This choice if made opens up the door to even more possibilities: if rolling is configured exactly like testing but with only a subset of the architectures, it s likely that some packages migrate to rolling before testing when non-mainstream architectures are lagging in terms of auto-building (or have toolchain problems). While being ahead of testing can be positive for the users, it s also problematic on several levels. First, managing rolling becomes much more complicated because the transition management work done by the release team can t be reused as-is. Then it introduces competition between both distributions which can make it more difficult to get a stable release out, for example if maintainers stop caring of the migration to testing once the migration to rolling has been completed. The rolling distribution is certainly a good idea but the rules governing it must be designed to avoid any conflict with the process of releasing a stable distribution. Lastly, the mere existence of rolling would finally fix the marketing problem plaguing testing: the name rolling does not suggest that the software is not yet ready for prime time. Conclusion Whether CUT will be implemented remains to be seen, but it s off for a good start: ftpmaster Joerg Jaspert said that the new archive server can cope with a new distribution, and there s now a proposal shaping up. The project might start quickly: there is already an implementation plan for the snapshot side of the project. The rolling distribution can always be introduced later, once it is ready. Both approaches can complement each other and provide something useful to different kind of users. The global proposal is certainly appealing: it would address the concerns of obsolescence of Debian s stable release by making intermediary releases. Anyone needing something more recent for hardware support can start by installing a cut and follow the subsequent releases until the next stable version. And users who always want the latest version of every software could use rolling after having installed a cut. From a user point of view, there are similarities with the mix of usual and long term releases of Ubuntu. But from the development side, the process followed would be quite different, and the constraints imposed by having a constantly usable distribution are stronger: any wide-scale change must be designed in a way that it can happen progressively in a transparent manner for the user.

This article was first published in Linux Weekly News. If you want to see more articles like this, join Flattr and click on the flattr button below every article that you like.

21 comments Support my work

Wellwell, the meeting is over since Sunday evening, but for some reasons I only was able to send the minutes today. Go get them over at the project list while they are still fresh, before another one snatches them away from you. They are basically a short summary of what we did or intend to do soon. I do like such kind of meetings, they usually help a lot, both in getting things done but also in lifting up my spirit and motivation to do something. And so I am actually looking forward to the springtime plan we have and hope it will work out nicely, and that we get all of the team together. The more, the better. :)

And there, another day come and passed, still no real breakage, only some small things here and there. We are doing something wrong, definitely. Ok, granted, providing everyone who has experimental in their sources.list with a free upgrade path due to us suddenly forgetting that apt actually really wants this NotAutomatic: yes flag in the release instead of just looking it up in our database, later on doing the same to all of the backports user (accidently, happens when you have identical codebases), does count enough for a small icecream. But we didn t have anything serious, which is great. Alex finished patching dak rm in a way that we can now actually close all bugs attached to a (source) package we remove and was thinking of enhancing it to actually also be able to close wnpp bugs attached to it (think of O: bugs for example). Mark had design work on g-p-s, which actually is how do we generate packages and sources and how to move the config sanely into a database . Sounds easy, but trust me, it isn t, when you consider all the special cases Debian has. He also jumped in helpfully whenever there was a question around some coding and stuff, at one time turning up with the insanity/genius/younameit code that allows us to transfer even more of dak.conf into the database. This gets us a good way along the way that should end up with a dak.conf consisting of only the database details and nothing else. My job today looked similar like yesterday. Merge code I got, eat, remove lots of values from dak.conf. This weekend besides totally missing out on Baklava, had us change our dak code quite a bit: 46 files changed, 982 insertions(+), 1060 deletions(-). Besides that we do have about 18 points on our meeting notes, a mail going off to one debian lists sometime the next days summarizing it a bit more.

Hello fellow developers, The summer is over but I m happy to announce that this year s Summer of Code at Debian has been better than ever! This is indeed the 4th time we had the privilege of participating in the Google Summer of Code and each year has been a little different. This year, 8 of our 10 students succeeded in our (very strict!) final evaluations, but we have reasons to believe that they will translate into more long-term developers than ever, all thank to you. The highlight this year has been getting almost all of our students at DebConf10. Thanks again this year to generous Travel Grants from the Google Open Source Team, we managed to fly in 7 of our students (up from 3!). You certainly saw them, presenting during DebianDay, hacking on the grass of Columbia, hacking^Wcheering our Debian Project Leader throwing the inaugural pitch of a professional baseball game or hacking^Wsun-tanning on the tr s kitsch Coney Island beach. Before I give the keyboard to our Students, I d like to tell you that it will be the pleasure and honor of Obey Arthur Liu (yours truly, as Administrator) and Bastian Venthur (as Mentor) to represent Debian at the Summer of Code 2010 Mentors Summit on 23-24 October 2010, at the Google Headquarters in Mountain View. Like last year, we expect many other DDs to be present under other hats. We will be having 2 days of unconference on GSoC and free software related topics. We all look forward to reporting from California on Planet and soc-coordination@l.a.d.o! All of our students had a wonderful experience, even if they couldn t come to DebConf, that is best shared in their own voice, so without further ado, our successful projects: Multi-Arch support in APT by David Kalnischkies, mentored by Michael Vogt apt-get install MultiArch does mostly work now as most code is already merged in squeeze, but if not complain about us at deity@l.d.o! Still, a lot left on the todo list not only in APT so let us all add MultiArch again to the Release Goals and work hard on squeezing it into wheezy. Debbugs Bug Reporting and Manipulation API by David Wendt Jr., mentored by Bastian Venthur Hello, I m David Wendt, and I went to Debconf10 to learn more about the development side of Debian. Having used it since the 9th grade, I ve been intimately familiar with many of Debian s internals. However, I wanted to see the developers and other Debian users. At DebConf, I was able to see a variety of talks from Debian and Ubuntu developers. I also got to meet with my mentor as well as the maintainer of Debbugs. Content-aware Config Files Upgrading by Krzysztof Tyszecki, mentored by Dominique Dumont Config::Model is now capable of manipulating files using shorter and easier to write models. Thanks to that, packagers may start experiment with creating upgrade models. Further work is needed to support more complicated config files Dominique Dumont is working on DEP-5 parser, I ll shortly start working on a cupsd config file parser.
The best thing about DebConf10 is that every person I talked with knew what I was doing. I had a mission to get some feedback on my project. Everybody liked the idea of making upgrades less cumbersome. On the other side, it was my first visit to United States, so I decided to go on a daytrip on my own (instead of staying inside the building, despite heat warnings). I had a chance to visit many interesting places like Ground Zero, the UN headquarters, Grand Central Terminal, Times square and Rockefeller Center that was a great experience. Hurd port and de-Linux-ization of Debian-Installer by J r mie Koenig, mentored by Samuel Thibault Debconf10 was great! Among other people working on the installer, I met Aur lien Jarno from the Debian/kFreeBSD team and we worked together on a cross-platform busybox package. Besides, the talks were very interesting and I ve filled my TODO-list for the year.
For instance I learned about the Jigsaw project of OpenJDK, and how Debian would be the ideal platform to experiment with it. More generally, some people think Debian could push Java 7 forward and I d like to see this happen. Smart Upload Server for FTP Master by Petr Jasek, mentored by Joerg Jaspert I must say that it was great time for me in NY, I ve met and talked and coded with people from ftp-master team like Torsten Werner who helped me to push the project a bit further and with some other people who were looking forward to release of the tool which I hope they will use quite soon. Everybody interested, everybody excited, really cool place and time. And I can t forget the Coney Island beach and stuff, lot of fun, lot of sun;) Aptitude Qt by Piotr Galiszewski, mentored by Sune Vuorela Currently, development branches support full features searching, viewing extended package s informations, performing cache and packages operations. Code and GUI still require a lot of work which will be continued. Informations about further progress could be found on aptitude mailing list and repository rss channel. Debian-Installer on Neo FreeRunner and Handheld Devices by Thibaut Girka, mentored by Gaudenz Steinlin For me, DebConf 10 started at the airport, where Sylvestre Ledru (whom I didn t know of before) was wearing a GSoC 2007 t-shirt, that is, given the circumstances, almost equivalent to say I m a hacker, I m going to DebConf 10 .
I ve spent my time at the conference attending various talks, hacking, meeting DDs and other hackers (amongst others, my co-mentor Per Andersson, Paul Wise, Julien Cristau, Christian Perrier, Cyril Brulebois, Martin Michlmayr, Colin Watson and Otavio Salvadores who I have to thank for his patience while dealing with my questions), chatting, cross-signing keys, rushing to finish eating before 7pm, getting sunburnt, sightseeing (thanks, Arthur, for the lightning-fast tour of Manhattan!), and so on. Debian Developers and community, we count on you. See you next year! (cross-posted to debian-devel-announce@l.d.o and soc-coordination@l.a.d.o)

Wanted to post this yesterday, but somehow didnt work out, so here we go now Another meeting day passed, another lot of commits into our repository, another day without entirely breaking the system. But wait, one more chance to break it today, maybe we finally manage. :) Yesterday we got volatile sorted out for Squeeze (Lenny stays like it is, seperate). Its now a suite on ftp-master, and we have all the neccessary information and workflow around it defined. At least so far that we can operate the suite, we don t define what gets into it, we just take that from the volatile team (or, I think, in future, from the SRMs) in a kind of textfile similar to how testing works. Besides having Alex unbreak our bts categorize script, Mark fixing up our BYHAND processing and implementing the volatile stuff I mentioned above, we also merged a patchset from Luca that allows us to provide the packages.debian.org service with a tree containing changelogs, copyright files, NEWS.Debian and README.Debian. Metadata basically. Which they extracted in the past, and that was taking up quite a lot of resources. Much easier if we provide it and they just use it. This is now done for both, the main archive as well as the backports archive. As soon as I upgrade the security archive it will also be done there, which should additionally enable packages.debian.org to display those files for all archives (run by FTPMaster). This update is scheduled for the upcoming week, DSA already gave me the needed bits and pieces on a host so I can do it without breaking security during the process. We currently export this metadata set only to the packages.debian.org host. We do want to provide it to more, but I am waiting for a reply from DSA as to which machine I should use for the open access to those files. When I have that reply it should be available for you all. As a sidenote from that: If you need to compute any data that has to do with the Debian archive (information out of source/binary packages or something around it) and if you do that with your own scripts - you are most likely doing it wrong. Talk to us. We might be able to help you extracting the data you need and keep it updated. Recent examples for that are the above changelogs, but also things like the package-file-mapping snapshot uses to easily determine which file belongs to which source/version, but also information like the Maintainers index, the Uploaders index and whatnot. The simple way is - ask us. We are processing a shitload of data, it might not be too complicated to let your set fall out somewhere at the side, but if you never ask you will never find out Today we start working on some internals of dak and will have some more discussion on agenda points like data.debian.org and the git source format. We already had some words on other points, but those are ones which will take a long time more to fix, like generally reworking the internals of our database to enable us having multiple archives with pretty different access levels inside one machine. (Think of security archive on ftpmaster host. You want to ensure embargoed security issues actually stay embargoed until they are released. Means a good set of neccessary changes inside the database to handle this. We have ideas, but getting them implemented will take a bit).

---

And of course we had been out for food too, helps to stay alive. Turkish restaurant. All nice and well - until they told us they have NO BAKLAVA. Args. Suckers. They ruined this year! WAAAAAAAAAAAAAAAAH. I wish them a windows OS for their live! If not worse! (Can it be worse than windows?) ;-)

I announced it some while ago on debian-project, we have a FTPMaster meeting this weekend. It started out with a little workout session View image before we started discussing some things from our agenda. In the middle of the discussion we took a little sidestep, looking for a victim to promote to ftpmaster. So we selected someone not attending, he can t run away screaming. And for that: Send your condolences over to Torsten Werner. Or maybe congratulations, your call :) Right now Mark is fixing up our byhand processing and Alex is doing some other Debian work, while I m merging patches from Luca. Thats it for now, sometime soon we are going to eat, after which we put some more discussions on our agenda. Still two more days to go :)

It has been a pretty exhausting time since my last blog post in my area of Debian land . If you followed the news you should have seen some of it. Last weekend there was the point release. For those I have the (good?) position to sit in the middle of the working queue, where the hardest part (sorting out which stuff to include, getting it all build, etc) is already done. Still, due to the amount of steps neccessary to take its usually a multi-hour thing. This one release was even slower, as we had to wait for hours until the package lists had all been built - unfortunately the thing writing that keeps berkeley db files of data around as cache - and uses the FULL path to the files as key. How stupidly annoying when the path changes during a server move. Ohwell. The other big part that got a tiny bit of press coverage was the move from www.backports.org over to backports.debian.org. This was actually waiting for me for a good longish while already (err, well, since well before DC9 this was decided to happen) but now I finally got myself moved. This turned out to be much more work than I ever feared (which is part of the reason why it took so long), but some numbers:

• At the start the code changes between the revision running on oldbpo compared to HEAD: 303 files changed, 21227 insertions(+), 468357 deletions(-) (There had been 1723 commits in the master branch since the last time bpo got it, thats about a quarter of all we have in master!)
• Database schema version oldbpo: 5, HEAD: 35
• sub-sub-sub-optimal damn old shell scripts for the cronjobs.

The db update alone took me a week to figure out which steps to do in which order. Good to have an automated system, if you the manage to have the updates run specific to the main archive. Ups. When I was done with the database (during which I managed to just erase all work done by formorer and rhonda on the webpage, ups^2) I had some code and config cleanups to do, and especially manage to get it all running with a new path layout. Much less complicated, but enough work to do, before I finally was far enough to get it all move so far that formorer could start testing it and we could turn off the old system. At this time I had to fix up our shell scripts for our cron jobs, but that (with one exception) was a nearly straight forward easy task. And on Wednesday I managed another goal - backports is fully running the master branch of dak, the exact same code as on ftp-master. The cronjobs are slightly different, as well as the config (of course, duh), but otherwise its the same. My current goal and the last things left on this move todo is to change the cronjobs so that debian as well as backports do run the SAME scripts, but their configs list which tools should be run when. Considering that my dinstall cron already contains something pretty much a state machine I think I m insane enough to get this in shell, but when I told that to Mark (you know, the second ftpmaster :) ), I heard something very much like vomiting (you know, 700km between us, this might mean something) and then the plea to use a real language for the task. Oh my. I like my shell. :)

---

There is one big black cloud on the horizon to see. I have to do something similar for security. They are at a dak (and database) version pretty similar to what bpo had. They do need to support this obnoxious spare new format too before they can support security for squeeze, so there is no way I can ignore that archive for much longer. I already asked DSA for a vm in which I can play (not good to interrupt security archive for long), but probably wont start on this before the upcoming FTPMaster meeting. Pretty busy otherwise. Wellwell, I hope it will be less painful, as I now know the steps already. :)

I just merged a nice work from another ftpteam member, Luca Falavigna. He wrote dak make-changelog, which is intended for generating the changelogs for stable point releases. I took the chance and asked if he can extend it a bit, which he did, so today I could merge that work. Besides that we are able to generate ChangeLogs between all suites we have, we now generate a ChangeLog on every time we import a new set for testing, listing only the changes done in that run, by listing the changelog entries from the package, or in case of removed packages just listing their name with the removed version. We keep 4 rotations of that ChangeLog, you can find it (with the next mirror push) on all Debian mirrors in the dists/testing/ directory. This should be especially interesting for those people following testing on their systems, to easily see what got changed. :) Short update: We now keep the ChangeLog files with a date based name, deleting files which are older than 2880 minutes, so roughly keeping 2 days or 4 testing runs around, but in case we import more often (hey, possible short before a release), we keep more. There is a symlink ChangeLog always pointing to the latest file.

* So, here is late report! Sometime back Praveen started discussion at DebianIndia mailing list about having MiniDebConf kind of conference here in India. MiniDebConfs are generally focused on development rather than at introduction to Debian . But, since this was our first attempt, out focus was more on to introduce Debian to student community and after writing this after 5 days I realize that we ve done good job! Many thanks to Joerg Jaspert for setting up Mini website: http://in2010.mini.debconf.org/ Praveen got bunch of sponsors, many thanks to CDAC and other sponsors for making event happen with testy food and tea that kept me awake. Now what really happened at Pune? My flight was nice. I went to guest house of COEP (the host college, 2nd old engineering college in India) and took some rest and called Praveen. He was already at computer lab and setting up Virtualbox and other things. We had two places hall and lab. Unfortunately, both were at different buildings and subway between them I discussed with couple of students and we went to dinner at late night. Pro. Abhijit who was kindly give permission to use lab and college hall was another source of energy. He is handling many FOSS events in college and probably only faculty who encourage students to use and learn FOSS. The Day 1 started with long lines of registration process. As we had around 200 registration, turn out of students were 100% on first day. Hall was almost full. Dr. Shashikumar from CDAC gave nice lecture about contribution which was well received. Praveen and Onkar then took the stage and gave nice introduction about Debian. I met Onkar first time. We had really nice discussion ranging from becoming Ubuntu MOTU to his Powerbook s adapter. After that Vikram took lively session on i18n and l10n in Debian and I took boring session on Contributing to Debian . Yet another boring session from me was Developer Toolbox . Packaging workshop by me at hall made many faces to yawn heavily while Onkar and Praveen handled it nicely at Lab. Main reason for yawning was Wifi and my Macbook s capacity to deal with ndiswrapper and crappy broadcom s drivers. (Note: I should get nice machine.) I met Sham first time. I met Kushal once again and felt energetic again! Day 2 As we expected number was very low compared to Day 1. So, we closed hall and moved entire stuffs to Lab only. Vikram took again nice session on Debian Edu . It was very interactive and I should learn something from his act of presentation. Another nice thing from Day 2 was discussion with BOSS Linux team. Hold on. Lets keep some old woes aside. They are basically doing what their bosses are telling. Developers were clueless about Sending patches to upstream. Their bugzilla had Zorro bug reported. I made promise to follow-up on this and set-up patches from boss at somewhere depending upon their response. That mail is my draft let me finish it by weekend. Pavi and others took video from session. It will be nice if someone will post it somewhere We then continued on discussing bugs and packaging area. It was amazing to see Praveen become DM during conference and also to see: upstream, maintainer and sponsor in one room. I really had nice food at dinners. Mango Mastani was nice. Beer was nice. How can I forget discussion with Ramki, Ninad, Pro.Abhijit, Praveen, Pavi and others after dinner? So lets work for MiniDebConf 2011 now!

I am currently sitting somewhere around Gate A2 at Singapore Airport, waiting for my connection flight. Not my original planned one. Not the one I got first rebooked to I was in Taiwan the last 3 weeks, having had about the best time ever. The plan for the flight back was pretty simple. Today, that is Tuesday 10th August, I take the Singapore Airlines flight from Taipei to Singapore, leaving Taipei at 1810h, arrival Singapore 22:35h. I wait a little time and then board the flight from Singapore to Frankfurt, leaving at 23:50h and touchdown in Frankfurt at 06:30h. So far so easy. So far not working at all. The flight from Taipei had technical problems (they later said problems with the computer of the engines ) and was delayed by a bit more than an hour. Not enough time to get the connection flight. So the ground staff in Taipei arranged a flight for me leaving Singapore at 13:50h the next day, Hotel and Taxi and such things to be organized by Singapore ground staff. Fine fine, so over to Singapore. Got here at 23:35h, left airplane at 23:45h. Definitely not working for the original connection, so the rebooking sounded good, When I left the airplane there was ground staff waiting for me, with the information that they got another reschedule done for me. I am now supposed to fly via Zurich, getting another connection from there to Frankfurt. So over to the transfer desk to get them to handle my luggage and actually get me a usable seat. Fine fine, all worked. Leaving the transfer desk, heading over to the gate (boarding meant to start at 00:10h, so not much time left), I have the pleasure to listen to an announcement: The flight to Zurich is delayed by approximately 2 hours. Oh fun, the planned connection flight in Zurich leaves about 2 hours after the planned arrival of this flight. Back to the transfer desk, asking whats up. Simple solution, I m supposed to fly to Zurich and contact the transfer desk there, it will deal with it. Lets see what happens and if I ever get back home. :) Nice side detail - I am now supposedly getting onto an A380. Lets see if this thing is really as great as they always say. Update: Now I am home. When we started in Singapore the Flight captain announced we have good weather helping us and will end up with being only 40 minutes late in Z rich. Wellwell, I think he just used up much more gas than usual, flying faster, but whatever, it worked out, 40 minutes late it was. (I cant believe always seeing 70mph of head wind being good ). So then I had to wait at that airport. And I must say: It sucks. Taipei airport - free wifi, just some funny redirections in browser before it works. Singapore - contact a staff member of the airport, get a code, works for 6 hours, fine. Z rich - the suckers want me to pay. A lot. Useless crapshit. Way to expensive. Bad service there. The next flight then finally got me to my flights destination, Frankfurt. Had to walk through half the airport to get to the luggage, but there it actually came out first. So fine. Train home after a short visit at my work was easy, and I think I will fall asleep soonish. Ohmy. Wish I would have stayed in Taipei

Thanks to work from Luca Falavigna we can now offer the Debian removals log in a slightly easier parseable format, RFC822. That means people can now follow removals from Debian in 3 formats:

• (halfway) life using RSS showing the last few removals done
• plain old easy TEXT, where the removals.txt directly shows the current year, previous years living in their own files:
  • 2009
  • 2008
  • 2007
  • 2006
  • 2005
  • 2004
  • 2003
  • 2002
  • 2001
    as well as having one FULL
• slightly easier parseable 822 format, with the same multitude as the text files, ie.
  • 2009
  • 2008
  • 2007
  • 2006
  • 2005
  • 2004
  • 2003
  • 2002
  • 2001
    as well as having one FULL

Not enough?

As SPI secretary I announced that nominations for the SPI board were open at the start of the month. The nomination period closes this Tuesday (13th July) with voting opening up on Thursday 15th. This year over half the board is up for election - 5 seats (currently held by Luk Claes, Joshua D. Drake, Bdale Garbee, Joerg Jaspert & Martin Zobel-Helas). So far I've received only 2 nominations, though I'm aware these things are often left to the last minute, so hopefully more will appear in the next few days. All anyone who wants to stand needs to do is drop secretary@spi-inc.org a (preferably PGP signed) email nominating yourself and providing a position statement (which will all be published once the nomination period is over).

Oh, and if you're a contributing SPI member please do remember to vote once voting is open!

As I wrote to d-d-a this morning, we are currently moving the ftp-master service from ries to franck. At the same time the release team moved their service over too. This was going on whole day, and we are still not done. There are bits and pieces we still have to check, but it is looking good. The new machine is also much much faster than the old one, for most of the tasks. For one it has 4 times the amount of CPU, double the RAM, but also something like a million times the disks. Well, not a million times, but it has much more storage on many many more disks. From the current state I think we will have the service fully back to normal level tomorrow, Sunday.

I d like to extend a warm welcome to our selected students for the 2010 Debian Google Summer of Code! They should pop up on Debian Planet soon and you re welcome to come talk to them on #debian-soc on irc.debian.org Aptitude Qt by Piotr Galiszewski, mentored by Sune Vuorela Qt GUI for aptitude. Currently, KDE users need to use Aptitude via the console interface, or install the newly developed GTK frontend, which does not fit well into KDE desktop. Making Qt frontend to Aptitude would solve this problem and bring an advanced and fully Debian-compliant graphical package manager to KDE. Content-aware Config Files Upgrading by Krzysztof Tyszecki, mentored by Dominique Dumont When a package deliver configuration files, the problem of merging user data with new configuration instructions will arise during package upgrades on users systems. Sometimes merging can be done with 3 way merge, but this process does not insure that the resulting file is correct or even legal. This project intends to create standards, tools an heuristics to make the scary config file conflict resolution debconf prompt a thing of the past. Debbugs Bug Reporting and Manipulation API by David Wendt Jr., mentored by Bastian Venthur Currently debbugs supports a SOAP interface for querying Debian s Bug Tracking System. Unfortunately this operation is read-only. This project would create an API for debbugs which supports sending and manipulating bug reports, without having to resort to email. This project does not intend to replace email as mean to manipulate the BTS but rather to enhance the BTS to allow other means of bug creation and manipulation. Debian High Performance Computing on Clouds by Dominique Belhachemi, mentored by Steffen Moeller The project paves a way to combine the demands in high performance computing with the dynamics of compute clouds with Debian. Combining the Eucalyptus cloud computing infrastructure with the TORQUE resource manager and preparing the components for dynamically added and removed instances provides the user with a attractive high performance computing environment. Such a system allows users to share resources with large compute centers with minimal changes in their workflow and scripts. Debian-Installer on Neo FreeRunner and Handheld Devices by Thibaut Girka, mentored by Gaudenz Steinlin This project aims to improve the installation experience of Debian on handheld devices by replacing ad-hoc install scripts by a full-blown and adapted Debian-Installer. The Neo FreeRunner is used as it is the most convenient and open device from a development standpoint, but other devices will also be explored. Hurd port and de-Linux-ization of Debian-Installer by J r mie Koenig, mentored by Samuel Thibault The primary means of distributing the Hurd is through Debian GNU/Hurd. However, the installation CDs presently use an ancient, non-native installer. The goal of this project is to port the missing parts of Debian-Installer to Hurd. To achieve this, all problematic Linux-specific code in Debian-Installer will be replaced by less or non-kernel dependent code, paving the way for better support of other non-Linux ports of Debian. Multi-Arch support in APT by David Kalnischkies, mentored by Michael Vogt Hardware like 64bit processors are perfectly able to execute 32bit opcode but until now this potentiality is disregard as the infrastructure tools like dpkg and APT are not able to install and/or solve dependencies across multiple architectures. The project therefore focuses on enabling APT to work out good solutions in a MultiArch aware environments without the need of hacky and partly working biarch packages currently in use. Package Repository Analysis and Migration Automation by Ricardo O Donell, mentored by Neil Williams Emdebian uses a filter to select packages from the main Debian repositories that are considered useful to embedded devices, excluding the majority of packages. The results of processing the filter are automated but maintaining the filter list is manual. This project seeks to automate certain elements of the filtering process to cope with specific conditions. This project will also generalize to more elaborate and intelligent algorithms to improve the transitions of the main Debian archives. Smart Upload Server for FTP Master by Petr Jasek, mentored by Joerg Jaspert Making packages upload smarter, more interactive and painless for uploaders by switching from anonymous FTP and Cron jobs to a robust protocol and modern package checking and processing daemon. This daemon would test early and report early, saving developers time. More details coming soon on http://wiki.debian.org/gsoc Congratulations everyone and have a fruitful summer!

Something that regularly happens when I get to see other peoples shell scripts is me thinking Why do they not use the powers of bash? . Sometimes it s fine to not use bash specific features, like when you are limited by the Operating System you have to run on (don t you try some commercial Unix systems. Or this WIndows crap. Brrr). Sometimes you want it for some perceived performance win, sometimes a full bash doesn t fit the target system (think embedded devices). Thats all fine, don t come up with those, I know them. What I mean is - people are using things that bind you to bash and your hashbang also makes your script use bash. But people still jump out to a different command just for some variable modification which are easily doable in bash too. A few examples follow, always imagine them in a script starting with the header of

#!/bin/bash
set -e
set -u

---

Example Get rid of newlines (actually, any replacement you can think of):

var1="something with
space and
linefeeds"
var1=$ var1//
/_ 
echo $var1

This is a simple but powerful demonstration of the Pattern substitution bash can do for you. Read its manpage, it can do much more than what I list here, but the above simply replaces all occurences of a linefeed with an _. Make the double // a single one and it is just the first that is replaced.

---

Example Removing a prefix (or suffix) from a string:

VAR1=$ VAR1##sync:archive: 

This removes the text sync:archive: from the beginning of the VAR1 variable. Replace the ## with %% and its gone from the end. The topic to look for in the manpage of bash is Remove matching prefix pattern (or suffix, of course).

---

There are many more possibilities, you might want to look through the Advanced Bash-Scripting Guide, it contains some helpful information. Or read the bash manpage. Or (shameless self-ad) look at the latest ftpsync script, the one and only official (mirroradm) supported mirror synchronization for the ftp.*.debian.org mirrors.

Another 422922 seconds (at the time of writing this entry) and another airplane down to Taiwan for me. This time direct flight, no KLM via Bangkok. 13 hours. Yikes. A two and a half week stay, then back up to Germany. Unlike last time I do not expect to fully disappear for the time. I actually expect to have some more time for my various jobs. Just in a way different timezone and mostly during that thing thats commonly called Office hours . Outside them, I dont think I will appear online much. Should you happen to be around Taipei and want to meetup - mail me. As long as its not during weekend it should be fine. :)